Running head:

Troubleshooting in a practice situation requires two types of information, namely for reasoning about the problem-cause and for finding an adequate solution (declarative information) and for manipulating the environment (procedural information). It is hypothesized that presenting this information piece-by-piece during practice (i.e., presentation of declarative and procedural information separately) frees up working memory and facilitates learning. Moreover, this effect is augmented when both information types are presented justin-time (i.e., declarative information before practice and procedural information during practice). This should yield highest test performance and instructional efficiency, which is defined as higher test performance combined with lower mental effort during practice. Eighty-five students (49 male, 36 female; M = 15.2 years, SD = .59) participated in a 2x2 factorial experiment with the factors timing of declarative information and timing of procedural information, both before or during practice. Transfer test scores and transfer efficiency scores support the first hypothesis; the second hypothesis was not supported. 3 Just-in-time Information Presentation Just-in-time Information Presentation: Improving Learning a Troubleshooting Skill Modern curricula make use of powerful e-learning environments for the acquisition of cognitive skills. Such environments contain practice problems (e.g., simulations) and related information (e.g., text and animation; see Merrill, 2002; Reigeluth, 1999; Van Merriënboer & Kirschner, 2001). During problem solving in such environments learners (1) master cognitive skills that require integration of the knowledge, skills, and attitudes necessary for effective performance, (2) learn to coordinate cognitive skills, and, eventually, (3) become able to transfer what is learned to their daily life or future work settings. Acquiring cognitive skills while working in complex learning environments, however, tends to cognitively tax the learner to such an extent that skill acquisition may be hampered or frustrated. Implementing guidelines from cognitive load theory (Sweller, 1988) could prove beneficial to facilitating the acquisition of cognitive skills in such environments. Cognitive load theory distinguishes three types of cognitive load, namely intrinsic, extraneous, and germane load. These three types of load need to be optimally balanced in order to make good use of limited working memory capacity (Baddeley, 1992; Miller, 1956). According to Sweller, van Merriënboer, and Paas (1998), intrinsic cognitive load is inherent to a learning task and is determined by the degree of element interactivity in that task. Extraneous cognitive load is caused by those processes a learner engages in while interacting with the instructional material that are not beneficial for learning. Examples of activities that induce extraneous cognitive load are mentally integrating different sources of information (e.g., separate information in a figure and a text) or searching for relevant information in order to understand the subject matter. Finally, germane cognitive load is load associated with processes that are beneficial for learning. Variability of learning tasks, for example, may stimulate learners to construct better cognitive schemata (Spiro, Coulson, Feltovich, & Anderson, 1988; Sweller et al., 1998) and can be considered, thus, to being germane to 4 Just-in-time Information Presentation learning. In general, a well designed learning environment should properly manage intrinsic load, minimize extraneous load, and optimize germane load within the boundaries of working memory capacity. The focus of this study is on managing intrinsic load and minimizing extraneous load in relation to the information needed to solve practice tasks and acquire troubleshooting skills. In this study, two types of information are distinguished, namely declarative and procedural. Declarative information refers to the conceptual model of how a domain is organized (see the textual information in the right half of Figure 1 for an example) and enables learners to construct cognitive schemata through elaboration (Reigeluth, 1983, 1999). The declarative information presented is gradually coupled to already existing, relevant cognitive schemata in long term memory of the learner. Elaboration of declarative information yields cognitive schemata that contain domain-general knowledge which is particularly useful when learners have to deal with unfamiliar problem situations. Such situations require interpretation of cognitive schemata, that is, different use of the same domain-general knowledge. In the study presented here, reasoning about the differences between various connections in electrical circuits (e.g., series or parallel) and their influence on the circuit and the properties of elements in the circuit is just such a situation. Procedural information refers to task-specific rules along with their related facts, principles, or concepts which are necessary for schema automation (see the textual information in the left half of Figure 1 for an example). This information enables learners to form productions through knowledge compilation (Anderson, 1996): The translation of procedural information into procedural knowledge (i.e., internalized rules). Knowledge compilation of procedural information yields productions containing domain-specific knowledge particularly useful for dealing with familiar problem situations because such situations require the same use of the same domain-specific knowledge. In the study presented 5 Just-in-time Information Presentation here, this allows the learner to recognize a switch and the need to throw it to close a circuit or to recognize a short circuit and be able to fix it. Intrinsic load can, first, be managed by presenting both the declarative and the procedural information piece-by-piece. Instead of allocating working memory capacity to processing both declarative and procedural information at the same time, learners can allocate the same amount of working memory capacity to the declarative and the procedural information, one piece at a time. In addition, intrinsic load can be managed by presenting the declarative information just-in-time, that is, before learners start solving the practice tasks. Since this declarative information typically has a higher degree of element interactivity (i.e., more interconnected elements) than the procedural information, presenting it during practice might require too much working memory capacity. When declarative information is presented before practice, all working memory capacity can be allocated to elaborating it and, thus, to schema acquisition. To minimize extraneous load, procedural information should also be presented just-intime, but here that means during practice-task solution. Extensive research has been carried out on the split attention effect (for an overview, see Sweller et al., 1998; for a study in this domain, see Kester, Kirschner, & van Merriënboer, in press). This research indicates that extraneous load is significantly reduced when mutually referring information resources are integrated, rather than when they are separated in time or space. Eliminating split attention in instructional material enables learners to allocate all available working memory capacity to processes relevant for learning. By presenting the procedural information fully integrated within the practice tasks, temporal and spatial split attention is avoided and all relevant information is active in working memory when it is applied to problem solving, a necessary precondition for knowledge compilation to occur (Kester, Kirschner, van Merriënboer, & Bäumer, 2001). 6 Just-in-time Information Presentation To investigate the effectiveness and efficiency of piece-by-piece and just-in-time information presentation, four information-presentation formats are compared, namely: 1. presenting declarative information before practice and procedural information during practice (i.e., piece-by-piece and just-in-time), 2. presenting declarative information during practice and procedural information before practice (i.e., piece-by-piece), 3. presenting both declarative and procedural information before practice, and 4. presenting both declarative and procedural information during practice. Effectiveness is measured by performance on two types of test tasks. Equivalent test tasks are tasks that are analogous to the practice tasks and, therefore, have a high level of familiarity to the learner. These tasks make use of the same circuit elements which were used during learning. Transfer test tasks are tasks that use some of the same elements used during learning along with new elements and have, thus, a lower level of familiarity. In such a transfer task, a motor would be used instead of a lamp to draw current in a circuit. Although learners need both specific schemata and general schemata to solve both types of test-tasks, solving equivalent tasks relies more on using specific schemata acquired through compilation of procedural information while solving transfer tasks increasingly relies on general schemata acquired through elaboration of the declarative information as the familiarity of these tasks decreases. While test performance indicates the effectiveness of the information presentation formats, the costs at which this performance is obtained indicates their efficiency. Instructional efficiency looks at the learning outcomes (i.e., test performance) in relation to the working memory capacity allocated during practice (Tuovinen & Paas, 2004; Paas, Tuovinen, Tabbers, & van Gerven, 2003; Paas & van Merriënboer, 1993). Instructional efficiency indicates the extent to which learners, during practice, were able to allocate their 7 Just-in-time Information Presentation working memory capacity to processes relevant for learning. High test performance in combination with low allocation of working memory capacity is highly efficient and indicates that the allocated working memory capacity was used for processes relevant to learning (i.e., low extraneous load). Low test performance in combination with high allocation of working memory capacity during practice is less efficient and indicates that a substantial amount of working memory capacity was allocated to processes not relevant to learning (i.e., high extraneous load). Performance efficiency provides information on the efficiency of the test performance, by looking at test performance in relation to the working memory capacity used to reach this performance. Performance efficiency indicates the extent to which learners were able to acquire adequate schemata during practice. High test performance in combination with low allocation of working memory capacity during the test is highly efficient and indicates that learners were able to form adequate schemata during practice to help them solve the test problems. Low test performance in combination with high allocation of working memory capacity during the test is less efficient and indicates that learners were not able to form adequate schemata during practice and possibly had to resort to more cognitively demanding, weak-method problem-solving strategies. The main hypothesis in this study is that the format where the declarative information is presented before practice and procedural during practice along with the format where the declarative information is presented during practice and procedural information before (i.e., piece-by-piece) yield a higher test performance, instructional efficiency, and performance efficiency than those formats that present both information types simultaneously, either before or during practice, because the piece-by-piece formats better manage intrinsic load. Moreover, it is hypothesized that the information presentation format that presents declarative information before practice and procedural information during practice (i.e., piece-by-piece 8 Just-in-time Information Presentation and just-in-time) will yield higher test performance, instructional efficiency, and performance efficiency than the other three formats because in this format the intrinsic load is properly managed and the extraneous load is minimized. Method Participants Eighty-five tenth-grade students at Bernardinus College, an academic high school in Heerlen, The Netherlands (49 male, 36 female; M = 15.2 years, SD = .59), participated in this study. All participants spoke Dutch as their first language, the language in which the instruction was given. They voluntarily participated in a physics lesson on electrical circuits, using a computer-based learning environment. No specific grade was given for this lesson. All participants followed the same physics education curriculum, which started in ninth grade. They were all equally familiar with the topic of the lesson because they all studied relevant theory in the previous academic year. As compensation for their participation they received a music compact disc of their own choice. Materials Physics lesson. Crocodile Physics 1.5, a simulation program for secondary school science classes, was used to develop the physics lesson for this experiment. The computerbased lesson contained an introduction, declarative information, procedural information, ten practice-troubleshooting tasks, and ten test-troubleshooting tasks. During practice, the participants had to use the declarative and procedural information given to troubleshoot a malfunctioning electrical circuit. They had to give a description of the problem, diagnose the cause of the problem, and present a solution to the problem. The aim of the lesson was to teach the participants to solve problems related to current (e.g., too high or too low), wrongly connected elements (e.g., lamps, switches, and meters in series or parallel), and short circuits. This was tested in ten test tasks where participants had to apply what they had learned during 9 Just-in-time Information Presentation practice to ten novel malfunctioning circuits. No declarative or procedural information was available during the test. Information presentation. The troubleshooting practice tasks, consisting of malfunctioning electrical circuits, were accompanied by declarative and procedural information presented either before practice, during practice, or before and during practice. The declarative information was divided over three screens while the procedural information fit on one screen. All conditions contained the exact same amount of information, both declarative and procedural. Four information presentation formats were distinguished in a 2 x 2 factorial design with the factors being: 'timing of declarative information presentation', before or during practice, and 'timing of procedural information presentation', also before or during practice. In the 'declarative before, procedural during' format the declarative information was presented before practice while the procedural information was presented during practice. In the 'declarative during, procedural before' format the declarative information was presented during practice while the procedural information was presented before practice. In the 'declarative before, procedural before' format both declarative information and procedural information were presented before practice. Finally, in the 'declarative during, procedural during' format both declarative and procedural information were presented during practice. For an example of a practice task see Figure 1. ***INSERT FIGURE 1 ABOUT HERE*** Practice tasks. Participants received ten practice tasks. The circuits in the practice tasks made use of six elements, namely: a toggle switch, a lamp, a battery, a resistor, a voltmeter, and an ammeter. During practice, all circuits contained all six elements. Every practice task consisted of three parts. The participants could obtain a maximum of 30 points by giving a correct problem description (1 point), a correct problem cause (1 point), and a correct problem solution (1 point) for the malfunctioning circuits. For example, the inserted 10 Just-in-time Information Presentation battery in one practice task is too strong for the elements in the circuit. In this task the following correct responses could be made: (1) problem description: the lamp explodes (= 1 point), (2) problem cause: the power supply, for instance the battery, is too strong for the lamp (= 1 point), and (3) problem solution: insert a weaker battery (= 1 point). The problem causes that were implemented in the practice tasks were related to current (i.e., too high or too low), wrongly connected elements or a short circuit. Every task contained only one problem. To determine interrater reliability, practice performance scores of eight participants were determined by two raters. The interrater reliability for practice performance of the two raters was .87 (Intraclass Correlation Coefficient, SPSS). The internal consistency of the practice items was .82 (Cronbach's alpha). Test tasks. After the ten practice tasks, participants received ten test tasks. The test tasks also consisted of malfunctioning electrical circuits with problems related to current, wrongly connected elements and short circuits that were designed in Crocodile Physics, but without the accompanying information. Five of the ten test tasks were equivalent to the practice tasks, that is, they contained the same elements as the practice tasks. The other five tasks also contained one or two new elements, namely: a variable resistor, a fuse, an LED, a buzzer and push-button switch, or a motor and gears. The equivalent test-tasks were meant to determine whether the participants had formed specific schemata to help them solve the familiar aspects of the test task. The test tasks that contained a new element, the transfer test-tasks, were meant to determine whether the participants were able to construct specific schemata plus those general schemata that help them solve the unfamiliar aspects of the test task. The participants could obtain a maximum of 15 points for the five equivalent tasks and 15 points for the five transfer tasks. As was the case in practice, they received one point for each correct description, cause or solution. To determine the interrater reliability the total test performance scores of eight participants were 11 Just-in-time Information Presentation determined by two raters. The interrater reliability for the test performance on the equivalent tasks of the two raters was .87 (Intraclass Correlation Coefficient, SPSS) and for the transfer tasks it was .92. The internal consistencies of the equivalent tasks and the transfer tasks were .64 and .63 (Cronbach's alpha), respectively. Mental effort measurement. Mental effort was used as an index of cognitive load. It refers to the amount of working memory capacity allocated to problem solving. Mental effort was measured during both practice and the test with a 9-point rating-scale (Paas, 1992; Paas, van Merriënboer, & Adam, 1994). The mental effort measures ranged from very, very low mental effort (1) to very, very high mental effort (9). The rating-scale was administered electronically during both practice and the test directly after each troubleshooting task. After each task, participants were asked: “How much mental effort did it require to find a solution for the problem(s) in the preceding circuit?” Moreover, after the ten practice tasks a separate mental effort measurement was administered for the subject matter. The participants were asked: “How much mental effort did it require to understand all subject matter?” The internal consistency of the mental effort measures was .83 (Cronbach's alpha) for the ten practice tasks, .69 for the five equivalent test tasks, and .68 for the five transfer test tasks. Learning and performance efficiency The Paas and van Merriënboer procedure (1993; see also Paas et al., 2003) was used to calculate efficiency. First, the performance measures and the mental effort measures for each participant are transformed into z-scores, using the grand mean across conditions. Then, the mean z-scores for every condition is represented in a Cartesian coordinate system with Mental effort z-scores on the horizontal axis and Performance z-scores on the vertical axis (see Figure 2). The line P = M through the origin indicates a neutral efficiency (slope = 45). The efficiency, E, is calculated as the perpendicular distance from a data point in the coordinate 12 Just-in-time Information Presentation system to the line P = M (Paas & van Merriënboer, 1993). The formula for calculating this distance is: Performance Mental Effort